12 research outputs found

    Domain-agnostic and Multi-level Evaluation of Generative Models

    Full text link
    While the capabilities of generative models heavily improved in different domains (images, text, graphs, molecules, etc.), their evaluation metrics largely remain based on simplified quantities or manual inspection with limited practicality. To this end, we propose a framework for Multi-level Performance Evaluation of Generative mOdels (MPEGO), which could be employed across different domains. MPEGO aims to quantify generation performance hierarchically, starting from a sub-feature-based low-level evaluation to a global features-based high-level evaluation. MPEGO offers great customizability as the employed features are entirely user-driven and can thus be highly domain/problem-specific while being arbitrarily complex (e.g., outcomes of experimental procedures). We validate MPEGO using multiple generative models across several datasets from the material discovery domain. An ablation study is conducted to study the plausibility of intermediate steps in MPEGO. Results demonstrate that MPEGO provides a flexible, user-driven, and multi-level evaluation framework, with practical insights on the generation quality. The framework, source code, and experiments will be available at https://github.com/GT4SD/mpego

    Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

    No full text
    A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings

    Sea Level Variability and Change

    Get PDF
    Land surface albedo represents the fraction of solar radiation scattered backward by land surfaces. In the presence of vegetation, surface albedo results from complex nonlinear radiation transfer processes determining the amount of radiation that is scattered by the vegetation and its background, transmitted through the vegetation layer, or absorbed by the vegetation layer and its background. Anomalies in mid- and high latitude regions of the Northern Hemisphere result mainly from interannual variations in snow cover extent and duration in winter and spring. The large negative anomalies over the United States reflect the lack of snowfall and snowpack over the Rockies, the Midwest, and much of the eastern half of the country.JRC.H.7-Climate Risk Managemen

    Meridional Overturning Circulation Observations in the Subtropical North Atlantic

    Get PDF
    Large-scale climate patterns influenced temperature and weather patterns around the globe in 2011. In particular, a moderate-to-strong La Niña at the beginning of the year dissipated during boreal spring but reemerged during fall. The phenomenon contributed to historical droughts in East Africa, the southern United States, and northern Mexico, as well the wettest two-year period (2010-11) on record for Australia, particularly remarkable as this follows a decade-long dry period. Precipitation patterns in South America were also influenced by La Niña. Heavy rain in Rio de Janeiro in January triggered the country's worst floods and landslides in Brazil's history. The 2011 combined average temperature across global land and ocean surfaces was the coolest since 2008, but was also among the 15 warmest years on record and above the 1981-2010 average. The global sea surface temperature cooled by 0.1°C from 2010 to 2011, associated with cooling influences of La Niña. Global integrals of upper ocean heat content for 2011 were higher than for all prior years, demonstrating the Earth's dominant role of the oceans in the Earth's energy budget. In the upper atmosphere, tropical stratospheric temperatures were anomalously warm, while polar temperatures were anomalously cold. This led to large springtime stratospheric ozone reductions in polar latitudes in both hemispheres. Ozone concentrations in the Arctic stratosphere during March were the lowest for that period since satellite records began in 1979. An extensive, deep, and persistent ozone hole over the Antarctic in September indicates that the recovery to pre-1980 conditions is proceeding very slowly. Atmospheric carbon dioxide concentrations increased by 2.10 ppm in 2011, and exceeded 390 ppm for the first time since instrumental records began. Other greenhouse gases also continued to rise in concentration and the combined effect now represents a 30% increase in radiative forcing over a 1990 baseline. Most ozone depleting substances continued to fall. The global net ocean carbon dioxide uptake for the 2010 transition period from El Niño to La Niña, the most recent period for which analyzed data are available, was estimated to be 1.30 Pg C yr-1, almost 12% below the 29-year long-term average. Relative to the long-term trend, global sea level dropped noticeably in mid-2010 and reached a local minimum in 2011. The drop has been linked to the La Nina conditions that prevailed throughout much of 2010-11. Global sea level increased sharply during the second half of 2011. Global tropical cyclone activity during 2011 was wellbelow average, with a total of 74 storms compared with the 1981-2010 average of 89. Similar to 2010, the North Atlantic was the only basin that experienced abovenormal activity. For the first year since the widespread introduction of the Dvorak intensity-estimation method in the 1980s, only three tropical cyclones reached Category 5 intensity level-all in the Northwest Pacific basin. The Arctic continued to warm at about twice the rate compared with lower latitudes. Below-normal summer snowfall, a decreasing trend in surface albedo, and aboveaverage surface and upper air temperatures resulted in a continued pattern of extreme surface melting, and net snow and ice loss on the Greenland ice sheet. Warmerthan- normal temperatures over the Eurasian Arctic in spring resulted in a new record-low June snow cover extent and spring snow cover duration in this region. In the Canadian Arctic, the mass loss from glaciers and ice caps was the greatest since GRACE measurements began in 2002, continuing a negative trend that began in 1987. New record high temperatures occurred at 20 m below the land surface at all permafrost observatories on the North Slope of Alaska, where measurements began in the late 1970s. Arctic sea ice extent in September 2011 was the second-lowest on record, while the extent of old ice (four and five years) reached a new record minimum that was just 19% of normal. On the opposite pole, austral winter and spring temperatures were more than 3°C above normal over much of the Antarctic continent. However, winter temperatures were below normal in the northern Antarctic Peninsula, which continued the downward trend there during the last 15 years. In summer, an all-time record high temperature of -12.3°C was set at the South Pole station on 25 December, exceeding the previous record by more than a full degree. Antarctic sea ice extent anomalies increased steadily through much of the year, from briefly setting a record low in April, to well above average in December. The latter trend reflects the dispersive effects of low pressure on sea ice and the generally cool conditions around the Antarctic perimeter. © 2012 American Meteorological Society

    State of the climate in 2010

    No full text
    Several large-scale climate patterns influenced climate conditions and weather patterns across the globe during 2010. The transition from a warm El Nino phase at the beginning of the year to a cool La Nina phase by July contributed to many notable events, ranging from record wetness across much of Australia to historically low Eastern Pacific basin and near-record high North Atlantic basin hurricane activity. The remaining five main hurricane basins experienced below-to well-below-normal tropical cyclone activity. The negative phase of the Arctic Oscillation was a major driver of Northern Hemisphere temperature patterns during 2009/10 winter and again in late 2010. It contributed to record snowfall and unusually low temperatures over much of northern Eurasia and parts of the United States, while bringing above-normal temperatures to the high northern latitudes. The February Arctic Oscillation Index value was the most negative since records began in 1950. The 2010 average global land and ocean surface temperature was among the two warmest years on record. The Arctic continued to warm at about twice the rate of lower latitudes. The eastern and tropical Pacific Ocean cooled about 1 C from 2009 to 2010, reflecting the transition from the 2009/10 El Nino to the 2010/11 La Nina. Ocean heat fluxes contributed to warm sea surface temperature anomalies in the North Atlantic and the tropical Indian and western Pacific Oceans. Global integrals of upper ocean heat content for the past several years have reached values consistently higher than for all prior times in the record, demonstrating the dominant role of the ocean in the Earth's energy budget. Deep and abyssal waters of Antarctic origin have also trended warmer on average since the early 1990s. Lower tropospheric temperatures typically lag ENSO surface fluctuations by two to four months, thus the 2010 temperature was dominated by the warm phase El Nino conditions that occurred during the latter half of 2009 and early 2010 and was second warmest on record. The stratosphere continued to be anomalously cool. Annual global precipitation over land areas was about five percent above normal. Precipitation over the ocean was drier than normal after a wet year in 2009. Overall, saltier (higher evaporation) regions of the ocean surface continue to be anomalously salty, and fresher (higher precipitation) regions continue to be anomalously fresh. This salinity pattern, which has held since at least 2004, suggests an increase in the hydrological cycle. Sea ice conditions in the Arctic were significantly different than those in the Antarctic during the year. The annual minimum ice extent in the Arctic reached in September was the third lowest on record since 1979. In the Antarctic, zonally averaged sea ice extent reached an all-time record maximum from mid-June through late August and again from mid-November through early December. Corresponding record positive Southern Hemisphere Annular Mode Indices influenced the Antarctic sea ice extents. Greenland glaciers lost more mass than any other year in the decade-long record. The Greenland Ice Sheet lost a record amount of mass, as the melt rate was the highest since at least 1958, and the area and duration of the melting was greater than any year since at least 1978. High summer air temperatures and a longer melt season also caused a continued increase in the rate of ice mass loss from small glaciers and ice caps in the Canadian Arctic. Coastal sites in Alaska show continuous permafrost warming and sites in Alaska, Canada, and Russia indicate more significant warming in relatively cold permafrost than in warm permafrost in the same geographical area. With regional differences, permafrost temperatures are now up to 2 C warmer than they were 20 to 30 years ago. Preliminary data indicate there is a high probability that 2010 will be the 20th consecutive year that alpine glaciers have lost mass. Atmospheric greenhouse gas concentrations continued to rise and ozone depleting substances continued to decrease. Carbon dioxide increased by 2.60 ppm in 2010, a rate above both the 2009 and the 1980-2010 average rates. The global ocean carbon dioxide uptake for the 2009 transition period from La Nina to El Nino conditions, the most recent period for which analyzed data are available, is estimated to be similar to the long-term average. The 2010 Antarctic ozone hole was among the lowest 20% compared with other years since 1990, a result of warmer-than-average temperatures in the Antarctic stratosphere during austral winter between mid-July and early September
    corecore